Inventory management through image and data integration

ABSTRACT

In a computer-implemented method, a computer program product, and a system, at least one processor obtains and decides a signal of decodable indicia to obtain information identifying an object made up of items. The processor obtains data comprising descriptive text characterizing the portion of the object, including quantitative inventory data related to the portion of the object. The processor displays the visual representation as a three dimensional image and the descriptive text, via an AR/VR device, generating a virtual projection in three dimensional space in a range of view of a user utilizing the device. The processor obtains a designation of a region in the visual representation and executes an action that changes a quantitative or a qualitative element of the descriptive text for an item represented by the region. The processor updates the descriptive text in the visual representation, to reflect the change.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. Non-Provisional applicationSer. No. 16/229,396 filed Dec. 21, 2018, entitled, “INVENTORY MANAGEMENTTHROUGH IMAGE AND DATA INTEGRATION,” which claims priority to U.S.Provisional Application No. 62/609,584 filed Dec. 22, 2017, entitled,“INVENTORY MANAGEMENT SYSTEM AND METHOD,” which is incorporated hereinby reference in its entirety.

FIELD OF INVENTION

The invention relates generally to an inventory management and controlsystem designed specifically to model and visualize data for use inenhancing and optimizing various processes utilized in the healthcareindustry. However, the system and methods described are adaptable acrossadditional industries.

BACKGROUND OF INVENTION

Today, healthcare providers of healthcare products often do not havevisibility to these products once they leave the manufacturing facility,thereby creating significant inefficiencies and increased costs in themarketplace. Both users and providers lack technologies that reducehuman error in identifying and counting inventory in a way thatminimizes contamination while improving efficiency. This lack ofinventory control equates to but is not limited to the followingindustry waste: 1) very low inventory turns equating to millions ofdollars in working capital waste; 2) high risk of expiration in thefield, creating risk that healthcare products could be surgicallyimplanted or dispensed in or to the human patient; 3) costly recall orfield action management, as inventory location is unknown once theproduct is shipped from the point of manufacture, and 4) an inventorysupply/distribution chain that is filled with un-necessary inventory,thereby driving the cost of product and product management upsignificantly and 5) limited or non-existent statistical tracking ofutilization of product while deployed at a facility.

One of the inefficiencies in inventory management is the process ofrecording inventory utilizing computer-readable indicia, such as barcodes and reading these codes with hand-held computing devices. Barcodes on the outside of inventory packages are encoded indicia thatreflect can be used to reflect original equipment manager (OEM) name,part number, lot and/or serial number, quantity, utilization, and/orexpiration date. Scanning bar codes on the outside of packaging using ahandheld scanning device can be time-consuming and introducespossibilities of user error when indicia are skipped.

SUMMARY OF INVENTION

Shortcomings of the prior art are overcome and additional advantages areprovided through the provision of a method for visualizing and modelingdata to manage inventory. The method includes: obtaining, by one or moreprocessors, a signal of decodable indicia; decoding, by the one or moreprocessors, the signal of decodable indicia to access decoded data,wherein the decoded data comprises information identifying an object,wherein the object comprises a plurality of items; based on theinformation identifying the object, obtaining, by the one or moreprocessors, from a memory, a visual representation of a portion of theobject, wherein the visual representation is divided into a plurality ofregions and each region represents an item of the plurality of items;based on identifying the object, obtaining, by the one or moreprocessors, data comprising descriptive text characterizing the portionof the object, wherein the descriptive text comprises quantitativeinventory data related to the portion of the object; displaying, by theone or more processors, the visual representation as a three dimensionalimage and the descriptive text, via a device, wherein the device isselected from the group consisting of: an augmented reality device and avirtual reality device, wherein the three dimensional image comprises avirtual projection in three dimensional space in a range of view of auser utilizing the device, wherein the device comprises a userinterface; obtaining, by the one or more processors, via the userinterface, a designation of at least one of the plurality of regions inthe visual representation; based on obtaining the designation, executingan action, wherein the action changes a quantitative or a qualitativeelement of the descriptive text for an item represented by the at leastone of the plurality of regions; and updating, by the one or moreprocessors, concurrently with the executing, the descriptive text in thevisual representation to reflect the changed quantitative or qualitativeelement.

Shortcomings of the prior art are overcome and additional advantages areprovided through the provision of a system for visualizing and modelingdata to manage inventory. The system includes, for example, a memory,one or more processors in communication with the memory, a device(selected from the group consisting of: an augmented reality device anda virtual reality device) communicatively coupled to the one or moreprocessors, program instructions executable by the one or moreprocessors via the memory to perform a method, the method comprising:obtaining, by the one or more processors, a signal of decodable indicia;decoding, by the one or more processors, the signal of decodable indiciato access decoded data, wherein the decoded data comprises informationidentifying an object, wherein the object comprises a plurality ofitems; based on the information identifying the object, obtaining, bythe one or more processors, from a memory, a visual representation of aportion of the object, wherein the visual representation is divided intoa plurality of regions and each region represents an item of theplurality of items; based on identifying the object, obtaining, by theone or more processors, data comprising descriptive text characterizingthe portion of the object, wherein the descriptive text comprisesquantitative inventory data related to the portion of the object;displaying, by the one or more processors, the visual representation asa three dimensional image and the descriptive text, via the device,wherein the three dimensional image comprises a virtual projection inthree dimensional space in a range of view of a user utilizing thedevice, wherein the device comprises a user interface; obtaining, by theone or more processors, via the user interface, a designation of atleast one of the plurality of regions in the visual representation;based on obtaining the designation, executing an action, wherein theaction changes a quantitative or a qualitative element of thedescriptive text for an item represented by the at least one of theplurality of regions; and updating, by the one or more processors,concurrently with the executing, the descriptive text in the visualrepresentation to reflect the changed quantitative or qualitativeelement.

Computer systems, computer program products and methods relating to oneor more aspects of the technique are also described and may be claimedherein. Further, services relating to one or more aspects of thetechnique are also described and may be claimed herein.

Additional features are realized through the techniques of the presentinvention. Other embodiments and aspects of the invention are describedin detail herein and are considered a part of the claimed invention.

BRIEF DESCRIPTION OF DRAWINGS

The foregoing and objects, features, and advantages of one or moreaspects of the invention are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 is an example of a technical architecture that can be utilized byembodiments of the present invention.

FIG. 2 is a component-level diagram of one embodiment of an EIRterminal, which is an example of a mobile computing device that can beutilized in an embodiment of the present invention with readingcapabilities.

FIG. 3 illustrates a block diagram of a resource in a computer system,which is part of the technical architecture of certain embodiments ofthe technique.

FIG. 4 is a computer program product that can execute program code toperform certain aspects of some embodiments of the present invention.

FIG. 5 is a technical environment into which aspects of some embodimentsof the present invention can be integrated.

FIG. 6 is a technical environment into which a workflow has beenintegrated in order to illustrate some aspects of some embodiments ofthe present invention.

FIG. 7 is an example of a composite image (also referred to as a datavisualization and/or projection) produced by program code in someembodiments of the present invention.

FIG. 8 is an illustration of some aspects of certain embodiments of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

Aspects of the present invention and certain features, advantages, anddetails thereof, are explained more fully below with reference to thenon-limiting examples illustrated in the accompanying drawings.Descriptions of well-known materials, fabrication tools, processingtechniques, etc., are omitted so as not to unnecessarily obscure theinvention in detail. It should be understood, however, that the detaileddescription and the specific examples, while indicating aspects of theinvention, are given by way of illustration only, and not by way oflimitation. Various substitutions, modifications, additions, and/orarrangements, within the spirit and/or scope of the underlying inventiveconcepts will be apparent to those skilled in the art from thisdisclosure.

Embodiments of the present invention significantly improve the abilityof consumers to manage inventory as well as to generate visuals of theinventory and provide real-time information to individuals makingdecisions regarding inventory. Aspects of the present invention performan improved service offering (greater functionality, greater informationvisibility, improved accuracy, improved business decision making) at amuch lower cost than the current industry solution for industries,including but not limited to, the medical and healthcare industry. Inaddition to financial efficiency, embodiments of the present inventionenable real-time re-ordering and real-time data capture andvisualization, to increase the technical efficiencies of embodiments ofthe present invention.

Embodiments of the present invention are inextricably tied to computingand provide significantly more than existing inventory management andproduct realization (real-time lookup and comprehension and applicationof data) approaches. Embodiments of the present invention areinextricably tied to computing based on their reliance on andutilization of computing devices, such as virtual reality and augmentedreality interfaces, in order to provide visualizations to users andallow users to comprehend and manipulate data related to inventorymanagement, in real-time. Additionally, the interconnectivity of theaugmented reality and virtual reality devices to various databasesacross differing inventory management systems, and to other relevantmedical and healthcare databases enables the diversity of thefunctionality. The interconnectivity is enabled by communications overcomputing networks. Embodiments of the present invention are alsoinextricably linked to computing because various aspects are directed toenhancing a user interfaces utilized to interact with the program codeexecuting on one or more processors.

Embodiments of the present invention provide significantly more thanexisting approaches to inventory management and comprehensive datacapture and visualization related to data capture by providing a handsfree, wearable augmented or virtual reality system that providevisibility to inventory to a user. Users can include, but are notlimited to inventory providers, manufacturers, and/or clinicians.Additionally, embodiments of the present invention are not limited, likeexisting systems, to receiving inputs via barcode scanning, alone. Inembodiments of the present invention, various aspects enable programcode executing on one or more processing devices to obtain data viavisual inputs, such as bar codes, form factors, color, and/orelectromagnetic spectrum indicia including but not limited toradio-frequency identification (RFID), near-field communication NFC,and/or passive WiFi. Additionally, embodiments of the present inventioninclude aspects that are highly portable and therefore, can be utilizedin a variety of different environments.

Aspects of embodiments of the present invention enable users, includingbut not limited to, health care product providers and/or OriginalEquipment Manufacturers (OEMs) the ability of making inventory orobjects in the field highly visible. In some embodiments of the presentinvention, based on this visibility, a user can manipulate the objectsin order to complete tasks including but not limited to, obtainingadditional information about the objects and/or ordering additionalobjects. Visibility means that an authorized individual utilizing thepresent invention can locate or report on inventory or objects at anylocation or stocking location serviced for which there is data availableon a memory resources accessible to the system. This functionalityenables a single entity to manage the inventory or objects acrossmultiple physical locations because the inventory or item at eachlocation is “visible.” The term OEM refers to original manufacturers ofproducts or components that are purchased or consigned by anothercompany and retailed under that purchasing company's brand name. Theterm OEM refers to the company that originally manufactured the product.

Embodiments of the present invention can include, but are not limitedto: 1) an inventory management and control system; 2) a wearable and/orhand-held, electronic data capture hardware and software system withadvanced image recognition and indicia scanning and readingcapabilities; 3) an augmented and/or virtual reality device that isutilized by program code in embodiments of the present invention toprovide an interface for users; and 4) one or more programs comprisingan application programming interface (API) that handles communicationsbetween the inventory management system and the wearable device. In someembodiments of the present invention, the wearable device that producesthe AR/VR interface for the user also includes the advanced indiciaidentification capabilities. The wearable device can also include anoff-the-shelf AR/VR user interface, including but not limited to,commercially available products, such as those manufactured by Oculus.

As discussed above, in some embodiments of the present invention, bothvisual capabilities and data capture capabilities are integrated intothe wearable device. The wearable device can also include more than onedistinct device, communicatively coupled, in order to display theinventory as well as obtain data related to the inventory. The inventorymanagement system can be a proprietary system and/or a third partysystem. Thus, the API can broker communications between the inventorymanagement system and the AR/VR device, such that the API obtains datafrom the inventory management system, displays data in an originalmanner utilizing the advantages of the AR/VR device, in the interface ofthe AR/VR device, and obtains inputs from the user via the AR/VR device,in order to take actions related to the displayed inventory item(s).These actions can include, but are not limited to: 1) generating anorder request for an item represented in the AR/VRsimulation/visualization created by the program code executing on thedevice; 2) counting inventory for an item represented in the AR/VRsimulation/visualization created by the program code executing on thedevice; 3) triggering a recall of an item represented in the AR/VRsimulation/visualization created by the program code executing on thedevice; and/or 4) triggering a search in the one or more designateddatabases for additional data to supplement the descriptive text(generated by the program code and also displayed in the interface)related to an item represented in the AR/VR simulation/visualizationcreated by the program code executing on the device.

In an embodiment of the present invention, the gathering of inventory oritem information can be accomplished by utilizing a AR/VR computingdevice, including what is referred to, generally, herein, as an encodedinformation reading (EIR) terminal, configured to read bar codes and/orother types of encoded information, to scan indicia, including but notlimited to, bar codes, RFID, NFC and/or Passive WiFi on the packaging ofitems. Encoded indicia on packages can be used to reflect originalequipment manager (OEM) name, part number, lot and/or serial number,quantity, and/or expiration date or other data. As discussed earlier,individually scanning bar codes on the outside of packaging using ahandheld scanning device can be time-consuming and introducespossibilities of user error when indicia are skipped. Thus, anotheradvantage embodiments of the present invention offer over existingapproaches to inventory management is that in some embodiments of thepresent invention, the aforementioned mobile device can be utilized toscan a group of items at one time by utilizing software and/or hardwareto capture an image of a display of inventory or items and extract barcodes and/or other encoded indicia from the image and provide avisualization of data retrieved from a coupled data source, based on thescanning, via augmented reality devices. In some embodiments of thepresent invention, the method further includes utilizing software and/orhardware to decode the extracted bar codes or other information, and toretain the decoded information. The retained information assists inmanaging the inventory or items, both on the particular augmenteddisplay and/or at different locations.

FIG. 1 is an example of an architecture 100 that can be utilized byembodiments of the present invention. At least on computer server 110with at least one processor executes program code as part of aninventory management and control system. In some embodiments of thepresent invention, the at least one server 110 is one or more resourcesof a cloud computing system and/or other shared or distributed computingsystem. In some embodiments of the present invention, the program codeprovides the described functionality as a service to subscribers to theshared computing system. This program code is also referred to assoftware as well as one or more programs throughout this disclosure.

Memory resources on the computer server 110 and and/or in communicationwith the program code execute on the server and can also comprise partof an inventory management and control system. The server 110communicates with at least one mobile computing device 120, includingbut not limited to, an encoded information reading (EIR) terminal,including but not limited to an AR/VR device 170, configured to read barcodes and/or other types of encoded information and to provide an AR/VRdisplay and interface to enable a user to interact through thisinterface. In an embodiment of the present invention, the mobilecomputing device 120 comprises an EIR terminal with image recognitionand bar code scanning and electronic indicia reading capabilities, whichare utilized, in an embodiment of the present method, for accuratelyinventorying healthcare products, including but not limited to, medicaldevices. (As explained earlier, enhanced inventory management as itpertains to the healthcare industry is discussed through to provide anon-limiting and concrete example of the applicability of aspects of thepresent invention to a particular field. However, as understood by oneof skill the art, aspects of some embodiments of the present inventioncan be implemented across other industries to enable enhanced inventoryvisualization and management.)

In FIG. 1, the server 110 can communicate with the mobile computingdevice 120 over a communications connection 130. The AR/VR device 170can similarly communicate with the mobile computing device 120 over acommunications connection 130. As understood by one of skill in the art,the EIR reading capability, which in this example is integrated into themobile computing device 120, can also be integrated into an AR/VR device170. In embodiments of the present invention, AR/VR devices 170 includewearable, implanted, handheld, and/or stationary computing resources. Insome embodiments of the present invention, a user can utilize one ormore AR/VR devices 170 at the same time. Thus, in some embodiments ofthe present invention, the AR/VR device 170 has the dual function ofcapturing a signal of decodable indicia from an item and displaying dataobtained about the item, to the user, based on decoding the signal. Insome embodiments of the present invention, the program code obtains datarelated to an item, based on the appearance of the item (no decoding isdone) and is able to display a visualization of the item in the AR/VRinterface for manipulation by the user.

The at least one mobile computing device 120 may include an integratedand/or separate (as displayed in FIG. 1) augmented or virtual realitydevice 170. In embodiments of the present invention, the AR/VR device170 superimposes an image (generated by the device and/or an externalprocessor communicatively coupled to the device 170) on a user's view(in the field of view of the user in three dimensional space), and/or adisplay of a two dimensional and/or three dimensional image on themobile computing device 120. Thus, the AR/VR device 170 provides acomposite view. In embodiments of the present invention where theaugmented or virtual reality device 170 is a virtual reality device, theAR/VR device 170 generates and displays a three-dimensional image/dataor environment that can be interacted with by a user in a seemingly realor physical way. The device may comprise stationary and/or portablewearable/handheld electronic equipment. In embodiments of the presentinvention, the augmented or virtual reality device 170 can be wearable.In some embodiments of the present invention, a user interacts with theaugmented or virtual reality device 170 through (user) interfaces in thedevices or communicatively coupled to the devices, including but notlimited to, a haptic feedback interface, a voice recognition and/oraudio interface, a gesture recognition interface, and/or a biometricrecognition interface.

As understood by one of skill in the art, embodiments of the presentinvention can be utilized in a variety of environments to determineinventory and/or provide interactive experiences to users with variousitems. One non-limiting example of an environment where aspects of thepresent invention can be implemented is in a physical space 140 (e.g., asecure room, a hospital, a physically or electronically bounded area)where a specific type of item is kept. Although represented as a singleroom in FIG. 1, the room 130 represents one to many rooms and/or spaces.These spaces can be physically or virtually defined. Within the space140, one or more programs executing on a processing device, in anembodiment of the present invention can capture, including via RFIDcapture, inventory passing into the space 140 and inventory passing outof the space 140. To obtain this data, a user can utilize the readingcapabilities of a wearable device of the present invention and/or theone or more programs can communicate with a device in the space 140,including but not limited to, an RFID reading devices 150 in the space140. The data captured by the RFID reading devices 150 can becommunicated electronically to the server 110. One or more deviceswithin the space that the program code can communicate with can includeInternet of Things (IoT) devices within the space both temporarily(e.g., worn by individuals who enter the space) or permanently situatedin the space (e.g., environmental sensors).

In FIG. 1, the physical space 140 is given as an example of merely onesource of data and one way in which the program code can delineate dataobtained and utilized in inventory management. In embodiments of thepresent invention, the mobile computing device 120 and/or the AR/VRdevice 170 can obtain data about items managed and displayed by theAR/VR device 170 for comprehension and manipulation based on obtainingdata from Internet of Things (IoT) devices, which are not necessarilyconstricted to a given physical setting. As understood by one of skillin the art, the Internet of Things (IoT) is a system of interrelatedcomputing devices, mechanical and digital machines, objects, animalsand/or people that are provided with unique identifiers and the abilityto transfer data over a network, without requiring human-to-human orhuman-to-computer interaction. These communications are enabled by smartsensors, which include, but are not limited to, both active and passiveradio-frequency identification (RFID) tags, which utilizeelectromagnetic fields to identify automatically and to track tagsattached to objects and/or associated with objects and people. Smartsensors, such as RFID tags, can track environmental factors related toan object or an area, including but not limited to, temperature andhumidity. The smart sensors can be utilized to measure temperature,humidity, vibrations, motion, light, pressure and/or altitude. IoTdevices also include individual activity and fitness trackers, whichinclude (wearable) devices or applications that include smart sensorsfor monitoring and tracking fitness-related metrics such as distancewalked or run, calorie consumption, and in some cases heartbeat andquality of sleep and include smartwatches that are synced to a computeror smartphone for long-term data tracking. Because the smart sensors inIoT devices carry unique identifiers, a computing system thatcommunicates with a given sensor can identify the source of theinformation. Within the IoT, various devices can communicate with eachother and can access data from sources available over variouscommunication networks, including the Internet. IoT devices can also beconfigured to communicate over private and/or secure networks withcomputing resources executing program code implementing aspects of someembodiments of the present invention. Additionally, certain IoT devicescan also be placed at various locations and can provide data based inmonitoring environmental factors at the locations. In some embodimentsof the present invention device 150 can be understood as an IoT deviceat a given location, which can be accessed by one or more programs in anembodiment of the present invention in order to collect data related toitems in the physical space 140.

FIG. 2 is a component-level diagram of one embodiment of an EIRterminal, which is an example of a mobile computing device 120 or acomponent of a mobile computing device 120 and/or AR/VR device 170 thatcan be utilized in an embodiment of the present invention. As discussedabove, the functionality of the EIR terminal in obtaining and decodingsignals of decodable indicia can be integrated into the AR/VR deviceutilized to visualize the data. In the example of FIG. 2, the EIRterminal 100 can comprise at least one microprocessor 310 and a memory320, both coupled to the system bus 370. The microprocessor 310 can beprovided by a general purpose microprocessor or by a specializedmicroprocessor (e.g., an ASIC). In one embodiment, EIR terminal 100 cancomprise a single microprocessor which can be referred to as a centralprocessing unit (CPU). In another embodiment, EIR terminal 100 cancomprise two or more microprocessors, for example, a CPU providing someor most of the EIR terminal functionality and a specializedmicroprocessor performing some specific functionality. A skilled artisanwould appreciate the fact that other schemes of processing tasksdistribution among two or more microprocessors are within the scope ofthis disclosure.

EIR terminal 100 can further comprise a communication interface 340communicatively coupled to the system bus 370. In one embodiment, thecommunication interface can be provided by a wireless communicationinterface. The wireless communication interface can be configured tosupport, for example, but not limited to, the following protocols: atleast one protocol of the IEEE 802.11/802.15/802.16 protocol family, atleast one protocol of the HSPA/GSM/GPRS/EDGE protocol family, TDMAprotocol, UMTS protocol, LTE protocol, and/or at least one protocol ofthe CDMA/1×EV-DO protocol family.

EIR terminal 100 can further comprise a keyboard interface 354 and adisplay adapter 355, both also coupled to the system bus 370. EIRterminal 100 can further comprise a battery 356. In one embodiment, thebattery 356 can be provided by a replaceable rechargeable battery pack.

EIR terminal 100 can further comprise a GPS receiver 380. EIR terminal100 can further comprise at least one connector (note pictured)configured to receive a subscriber identity module (SIM) card.

EIR terminal 100 can further comprise one or more EIR devices 330,provided, for example, but not limited to, by an RFID reading device, abar code reading device, or a card reading device. In one embodiment,EIR terminal 100 can be configured to read an encoded message using EIRdevice 330, and to output raw message data containing the encodedmessage. In another embodiment, the EIR terminal 100 can be configuredto read an encoded message using EIR device 330, and to output decodedmessage data corresponding to the encoded message. As used herein,“message” is intended to denote a character string comprisingalphanumeric and/or non-alphanumeric characters. An encoded message canbe used to convey information, such as identification of the source andthe model of a product, for example, in a UPC code.

Mobile computing devices that read bar codes, read RFID, or read cardsbearing encoded information can read more than one of these categorieswhile remaining within the scope of this disclosure. For example, adevice that reads bar codes may include a card reader, and/or RFIDreader; a device that reads RFID may also be able to read bar codesand/or cards; and a device that reads cards may be able to also read barcodes and/or RFID. For further clarity, it is not necessary that adevice's primary function involve any of these functions in order to beconsidered such a device; for example, a cellular telephone, smartphone,or PDA that is capable of reading bar codes is a device that read barcodes and can constitute a mobile computing device 120 and/or an AR/VRdevice 170, as seen in FIG. 1, for purposes of this disclosure.

In an embodiment of the present invention, the EIR terminal 100 includesa digital camera component 350, which enables the EIR terminal 100 tocapture digital images, which may be processed by the one or moreprocessing units of the EIR terminal 100. In embodiments of the presentinvention where the program code displays images and/or item datacaptured and/or decoded by the EIR terminal 100 on an interface of theAR/VR device (FIG. 1, 170). For example, as will be discussed later ingreater detail, the GUI of the AR/VR device (FIG. 1, 170) can display tothe operator that an item that was scanned by the EIR terminal 100 hasexpired and/or has been recalled.

In some embodiments of the present invention, the EIR terminal mayinclude a display device 390, including but not limited to a virtualreality device and/or an augmented reality device. As discussed above,the EIR terminal and the display can also be separate and can becommunicatively coupled. An API in embodiments of the present inventioncan broker communications between the AR/VR device and the EIR terminal.However, in embodiments of the present invention where the display anddata collection and interpretation functionalities are integrated into acommon device, as in FIG. 4, the display device 390 can generate a threedimensional image that the user can interact with in some capacity. Forexample, when the display device 390 includes an augmented realitydevice, the EIR terminal 300 generates an image and utilizes the device390 to superimpose the image on a user's view of the real world, thusproviding a composite view. As discussed herein, the composite view canalso include data relevant to the item(s) being displayed to the user inthe view. When the display device 370 includes a virtual reality device,the device 390 generates a simulation of a three-dimensional image/dataor environment. The environment is perceived by the user as acceptinginputs as the device 390 includes stationary and/or portablewearable/handheld electronic equipment that can receive inputs into theEIR device 300 based on the actions and movements of a user experiencingthe virtual reality generated by the device 390. The device can providethe user with haptic feedback to indicate that an input provided throughan action or a movement was accepted.

As aforementioned, in some embodiments of the present invention, ratherthan being internal to an EIR device 300, a virtual and/or augmentedreality device is communicatively coupled to the EIR device 300. Inembodiments such as these, while the virtual reality device can serve asan input device to the EIR terminal 300, the augmented reality devicemay enhance a display on the EIR device itself and a component of theEIR terminal 300 may accept input from a user on an input/output deviceof the EIR terminal 300 and/or communicatively coupled to the EIRterminal 300.

FIG. 3 illustrates a block diagram of a resource 200, like server 110,in computer system 100, which is part of the technical architecture ofcertain embodiments of the technique. As aforementioned, the computingresources in embodiments of the present invention can be part ofdistributed or shared computing environments, including cloud computingenvironments. The resource 200 may include a circuitry 202 that may incertain embodiments include a microprocessor 204. The computer system200 may also include a memory 206 (e.g., a volatile memory device), andstorage 208. The storage 208 may include a non-volatile memory device(e.g., EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, firmware, programmablelogic, etc.), magnetic disk drive, optical disk drive, tape drive, etc.The storage 208 may comprise an internal storage device, an attachedstorage device and/or a network accessible storage device. The system200 may include a program logic 210 including code 212 that may beloaded into the memory 206 and executed by the microprocessor 204 orcircuitry 202.

In certain embodiments, the program logic 210 including code 212 may bestored in the storage 208, or memory 206. In certain other embodiments,the program logic 210 may be implemented in the circuitry 202.Therefore, while FIG. 2 shows the program logic 210 separately from theother elements, the program logic 210 may be implemented in the memory206 and/or the circuitry 202.

Using the processing resources of a resource 200 to execute software,computer-readable code or instructions, does not limit where this codeis can be stored. The terms program logic, code, and software are usedinterchangeably throughout this application.

Referring to FIG. 4, in one example, a computer program product 400includes, for instance, one or more non-transitory computer readablestorage media 402 to store computer readable program code means or logic404 thereon to provide and facilitate one or more aspects of thetechnique.

As will be appreciated by one skilled in the art, aspects of thetechnique may be embodied as a system, method or computer programproduct. Accordingly, aspects of the technique may take the form of anentirely hardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,aspects of the technique may take the form of a computer program productembodied in one or more computer readable medium(s) having computerreadable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium or an augmented realityreadable signal or storage medium. A computer readable signal medium mayinclude a propagated data signal with computer readable program codeembodied therein, for example, in baseband or as part of a carrier wave.Such a propagated signal may take any of a variety of forms, including,but not limited to, electro-magnetic, optical or any suitablecombination thereof. A computer readable signal medium may be anycomputer readable medium that is not a computer readable storage mediumand that can communicate, propagate, or transport a program for use byor in connection with an instruction execution system, apparatus ordevice. A augmented reality medium may be a computer readable mediumthat can take a variety of forms and that is not a computer readablesignal or storage medium with computer readable program code embodiedtherein that can communicate, propagate or transport a program for useby or in connection with an instruction execution system apparatus ordevice.

A computer readable storage medium may be, for example, but not limitedto, an electronic, magnetic, optical, electromagnetic, infrared orsemiconductor system, apparatus, or device, or any suitable combinationof the foregoing. More specific examples (a non-exhaustive list) of thecomputer readable storage medium include the following: an electricalconnection having one or more wires, a portable computer diskette, ahard disk, a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM or Flash memory), anoptical fiber, a portable compact disc read-only memory (CD-ROM), anoptical storage device, a magnetic storage device, or any suitablecombination of the foregoing. In the context of this document, acomputer readable storage medium may be any tangible medium that cancontain or store a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing an appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thetechnique may be written in any combination of one or more programminglanguages, including but not limited to an object oriented programminglanguage, such as Java, Smalltalk, C++ or the like, and conventionalprocedural programming languages, such as the “C” programming language,assembler or similar programming languages. The program code may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider).

Aspects of the technique are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices including augmented realitydevices to function in a particular manner, such that the instructionsstored in the computer readable medium produce an article of manufactureincluding instructions which implement the function/act specified in theflowchart and/or block diagram block or blocks.

The computer program instructions, also referred to as computer programcode, may also be loaded onto a computer, other programmable dataprocessing apparatus, or other devices including augmented realitydevices to cause a series of operational steps to be performed on thecomputer, other programmable apparatus or other devices to produce acomputer implemented process such that the instructions which execute onthe computer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block and system diagrams in FIGS. 1-7 and FIG. 9illustrate the architecture, functionality, and operation of possibleimplementations of systems, methods and computer program productsaccording to various embodiments of the technique. In this regard, eachblock in the flowchart or block diagrams may represent a module,segment, or portion of code, which comprises one or more executableinstructions for implementing the specified logical function(s). Itshould also be noted that, in some alternative implementations, thefunctions noted in the block may occur out of the order noted in thefigures. For example, two blocks shown in succession may, in fact, beexecuted substantially concurrently, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. It will also be noted that each block of the block diagramsand/or flowchart illustration, and combinations of blocks in the blockdiagrams and/or flowchart illustration, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts, or combinations of special purpose hardware and computerinstructions.

In addition to the above, one or more aspects of the technique may beprovided, offered, deployed, managed, serviced, etc. by a serviceprovider who offers management of customer environments usingaugmented/virtual reality devices. For instance, the service providercan create, maintain, support, etc. computer code and/or a computerinfrastructure that performs one or more aspects of the technique forone or more customers using augmented/virtual reality devices. Inreturn, the service provider may receive payment from the customer undera subscription and/or fee agreement, as examples. Additionally oralternatively, the service provider may receive payment from the sale ofadvertising content to one or more third parties. Is payment usuallyaddressed in a patent?

In one aspect of the technique, an application may be deployed forperforming one or more aspects of the technique. As one example, thedeploying of an application comprises providing computer infrastructureoperable to perform one or more aspects of the technique.

As a further aspect of the technique, a computing infrastructure may bedeployed comprising integrating computer readable code into a computingsystem, in which the code in combination with the computing system iscapable of performing one or more aspects of the technique. As a furtheraspect of the technique, the system can operate in a peer to peer modewhere certain system resources, including but not limited to, one ormore databases, is/are shared, but the program code executable by one ormore processors is loaded locally on each computer, including the AR/VRdevice 170 of FIG. 1.

As yet a further aspect of the technique, a process for integratingcomputing infrastructure comprising integrating computer readable codeinto a computer system may be provided. The computer system comprises acomputer readable medium, in which the computer medium comprises one ormore aspects of the technique. The code in combination with the computersystem is capable of performing one or more aspects of the technique.

Further, other types of computing environments can benefit from one ormore aspects of the technique. As an example, an environment may includean emulator (e.g., software or other emulation mechanisms), in which aparticular architecture (including, for instance, instruction execution,architected functions, such as address translation, and architectedregisters) or a subset thereof is emulated (e.g., on a native computersystem having a processor and memory). In such an environment, one ormore emulation functions of the emulator can implement one or moreaspects of the technique, even though a computer executing the emulatormay have a different architecture than the capabilities being emulated.As one example, in emulation mode, the specific instruction or operationbeing emulated is decoded, and an appropriate emulation function isbuilt to implement the individual instruction or operation.

In an emulation environment, a host computer includes, for instance, amemory to store instructions and data; an instruction fetch unit tofetch instructions from memory and to optionally, provide localbuffering for the fetched instruction; an instruction decode unit toreceive the fetched instructions and to determine the type ofinstructions that have been fetched; and an instruction execution unitto execute the instructions. Execution may include loading data into aregister from memory; storing data back to memory from a register; orperforming some type of arithmetic or logical operation, as determinedby the decode unit. In one example, each unit is implemented insoftware. For instance, the operations being performed by the units areimplemented as one or more subroutines within emulator software.

Further, a data processing system suitable for storing and/or executingprogram code is usable that includes at least one processor coupleddirectly or indirectly to memory elements through a system bus. Thememory elements include, for instance, local memory employed duringactual execution of the program code, bulk storage, and cache memorywhich provide temporary storage of at least some program code in orderto reduce the number of times code must be retrieved from bulk storageduring execution.

Input/Output or I/O devices (including, but not limited to,augmented/virtual reality devices, keyboards, displays, pointingdevices, DASD, tape, CDs, DVDs, thumb drives and other memory media,etc.) can be coupled to the system either directly or throughintervening I/O controllers. Network adapters may also be coupled to thesystem to enable the data processing system to become coupled to otherdata processing systems or remote printers or storage devices throughintervening private or public networks. Modems, cable modems, andEthernet cards are just a few of the available types of networkadapters.

As aforementioned, an embodiment of the present invention includes oneor more mobile computing devices, such the augmented/virtual realitydevices (e.g., FIG. 1, 170) either communicatively coupled to orintegrated with an EIR terminal (e.g., FIG. 2). In an embodiment of thepresent invention, the mobile computing device is a wearable orhand-held, electronic data capture system with advanced imagerecognition and/or bar code, RFID, NFC, Passive WiFi, scanning andreading capabilities for accurately identifying items, including but notlimited to, counting healthcare products. In some embodiments of thepresent invention, the mobile computing device may be a virtual realitydevice and/or many be communicatively coupled to an augmented realitydevice. The virtual reality or augmented reality device may be utilizedto provide users with visuals that are immersive and three dimensionaland in some cases, enable the users to interact with the virtualrepresentation through movement and/or voice commands. In someembodiments of the present invention, the user may be alerted to theacceptance of inputs by the system through haptic feedback experiencedvia the augmented reality device or the virtual reality device.

One non-limiting example for use of aspects of some embodiments of thepresent invention is to order specific items from a collection of items.This non-limiting example is provided for illustrative purposes, only.In some embodiments of the present invention, the aforementioned deviceinterfaces with an inventory management system, also understood as adata manager to support inventory management and inventory replenishmentof products, including but not limited to, healthcare products and/ormedical devices. In an embodiment of the present invention, the programcode executed by a processing resource on the mobile device isconfigured for use in the medical device and/or healthcare industries.Among the software features customized for this industry is thecapability to select areas on an image of a device tray, where small,un-marked parts would be positioned and create a replenishment orderbased on this selection. In order to utilize this feature, a user canutilize the EIR capabilities of the mobile device to scan a barcodeand/or an RFID or other readable indicia and/or capture an image thatcan be recognized by the program code. After making this selection,program code executing either on the device and/or at the server decodesthis identification information to recognize the item that was scanned.In this example, the scanned item comprises a group of trays containingmany small items. After identifying the item scanned, the program coderenders a visual in a manner that provides a user with a threedimensional and potentially interactive view of each tray of the device.The program code utilizes the unique features of the AR/VR device (e.g.,FIG. 1, 170), to render the item in a composite view, which is discussedherein in FIG. 7.

In some embodiments of the present invention, the one or more programsmay renders the visual through a graphical user interface (GUI) on themobile device. Rather than view the visual directly in the GUI, the userutilizes an augmented reality device (e.g., a wearable device) to view athree dimensional representation of each tray. The augmented realitydevice overlays a three dimensional visual on the GUI, which is visibleto a user who wears the augmented reality device. In some embodiments ofthe present invention, only the augmented reality device and not the GUIrender the visual. In some embodiments of the present invention, themobile device is communicatively coupled to a virtual reality device orincludes an embedded virtual reality device. In these embodiments, thevirtual reality device not only renders a three dimensional image, thedevice also includes inputs that receive commands from the user, relatedto the image. For example, the virtual reality device may includesensors that track the user's movement and interpret certain movementsas certain inputs. Upon acceptance of the inputs, the virtual realitydevice may provide the user with haptic feedback.

Returning to the non-limiting example, viewing the visual, the usercompares the images of the trays to the actual trays and selects theregions on the image that represent missing items in the trays. The usermay select the missing items through a GUI, but while viewing theaugmented image, or by making movements or reciting voice commands thatare interpreted as specific inputs by the virtual reality device.Acceptance of the inputs may be indicated through the generation ofhaptic feedback by the device. The user selections are communicated fromthe mobile device to the data manager on the server and the data managercan then determine how many pieces there should be located in thatstorage slot and will issue a replenishment order for the missing numberof pieces.

Embodiments of the present invention, including those embodiments thatassist in managing inventory related, for example, to the medicalindustry, and, specifically, the healthcare industry, including but notlimited to, the medical device industry, utilize images of kits in aninteractive manner in order to assist in the replenishment of missingparts. Utilizing this technique, users (e.g., auditors) would select theitems from the image that are missing from the kit by utilizing variousinput methods, including but not limited to, touching the item in theimage, indicating the item with a voice command, and/or indicating theitem with a specific gesture (e.g., taps, gestures, blink/tracking,facial and hand movements). As understood by one or skill in the art, auser of an AR/VR device can utilize existing interfaces in communicatewith the program code. In an embodiment of the present invention,selecting elements of an image in the GUI would cause the program codeto “Gray Out” the selected item or items in the (augmented and/orvirtual) image. This graying of the product would signify that theitem(s) were not present in the kit. The program code would then buildthe inventory list based upon the image, meta-data and user inputs.

FIG. 5 is a technical environment 500 into which aspects of someembodiments of the present invention are integrated. FIG. 5 provides anexample of how aspects of the present invention work together, withinthis technical environment 500, to collect data, interpret data, andmanage inventory. As illustrated in FIG. 5, in some embodiments of thepresent invention, one or more programs executing on a processing deviceobtain, via an using augmented/virtual reality device 520 (illustratedin FIG. 5 as one or more of a wearable/implanted AR/VR projectioncomponent 520 and a wearable/implanted AR/VR glove 520) to capturevisual representations and decodable indicia of inventory objects useprogram code executing of a processor to identify objects, recall datafrom memory related to these objects (and/or obtain data from externalsystems based on executing queries), load data from memory, and displayselected data visually in the view of the user AR/VR device 520. In someembodiments of the present invention, the program code displays theadditional data related to the item adjacent to or near the subjectobject to create a composite view of the real object and the virtuallyvisualized relevant data.

As illustrated in FIG. 5, the program code identifies objects using avariety of signals and items of decodable indicia, and/or non-encodedcharacteristics, including but not limited to, RFID 501, a combinationof a barcode and RFID 507, NFC 502, passive WiFi 503, UHF/VHF 504, formfactor 505, and/or a combination of form factor, color, and texture 506.As indicated in FIG. 5, the readable indicia includes, but not limitedto, information broadcast via the Electromagnetic Spectrum, such as:visual data (e.g., bar code, form factor), RFID, and NFC, and audiblecues and/or combinations of indicia. The program code obtains this datavia one or more AR/VR devices 520. As illustrated in FIG. 5, the AR/VRdevice 520 captures the indicia from utilizing a readable indicia sensorarray/processor 517 that is either integrated into the AR/VR device 520,or communicatively coupled to the AR/VR device 520. The program code canproject the image, which can include a real and/or virtual image of theobject and data describing the object (based on utilizing the indicia toretrieve additional data from a data source), within a view of the user,as a projection 520, and/or as a handheld projection 508, on a computingdevice comprising the interface capable of the handheld projection 508.As discussed earlier, the user can interact with the projectionutilizing a variety of interfaces and the interactions are obtained bythe program code in embodiments of the present invention.

FIG. 6 is a technical environment 600 into which a workflow has beenintegrated in order to illustrate some aspects of some embodiments ofthe present invention. In this example, program code executing certainof the aspects of the invention executes on a processor of an AR/VRdevice 640. As illustrated in FIG. 6, because embodiments of the presentinvention include an API or other interfacing software/hardware (e.g.,middleware), the program code of embodiments of the present inventionprovides a communication interface between an inventory managementsystem 610 and an AR/VR device 630, such that various proprietary andthird party inventory management systems 610 and/or AR/VR devices 630can be utilized to perform aspects of some embodiments of the presentinvention. For example, inventory management systems utilized inembodiments of the present invention can include, but are not limitedto, WAN-based, LAN-based, and/or cloud-based computing resources. Forillustrative purposes, the exemplary inventory management system 610 ofFIG. 3 is a cloud computing system and includes resources such as amemory, a database processor, and/or analytical software.

In some embodiments of the present invention, a user wearing an AR/VRdevice 630, or otherwise utilizing it, perceives, in the user's field ofview, and, therefore, within a field of perception of the AR/VR device630, an inventory object 660 (609). The program code executing on theprocessor of the AR/VR device 640 obtains readable indicia (e.g., FIG.5) from the inventory object 660 (609) and identifies the inventoryobject 660 (601), based on the indicia. To identify the inventory object660, based in the indicia interpreted by the program code (601), theprogram code executing on the processor of the AR/VR device 640 queriesthe inventory management system 610 (602) and responsive to the query,which includes the indicia (including decoded indicia if the indicia wasencoded), obtains additional data related to the inventory object 660(604). For example, utilizing the example of the trays discussedearlier, responsive to identifying the object, the inventory object 660can provide the user with images that represent layers of a kit(assuming in this example that the kit is the identified inventoryobject 660). The data can also include recall information and/or datarelated to the amount of the inventory object 660 at a given locationand/or at a variety of locations. The data can also include statisticaldata related to the inventory object 660. The program code executing onthe processor of the AR/VR device 640 provides the additional data fromthe inventory management system 610 to the AR/VR device 640 (605). TheAR/VR device 640 generate a data projection 670 of both the objectitself (either keeping the object in the viewer and/or producing anenhanced image based on the additional data) and within the same view,provides the additional data. Thus, the (augmented reality and/orvirtual reality) device 640 provides a composite view to the user. Insome embodiments of the present invention, the composite view includes athree dimensional image of the object 660 and data related to the object660, including but not limited to, inventory count, and/or expirationdate. The program code can display the object 660 and/or the itemutilization and or data related to the item on the augmented/virtualreality device 640. The one or more programs can also generate avisualization which includes the data associated with the object orobjects displayed in augmented/virtual reality space, where the subjectobject or objects resides as observed by the user. In some embodimentsof the present invention, based on the type of data, the program codecan determine where/how it is visualized to the user.

Utilizing an interface provided by the AR/VR device 640 (and/orcommunicatively coupled to the AR/VR device 640), the user providesinput, based on the projection 670. Based on interacting with theprojection 670, the user experience can feel like the user is providinginput in the projection, while it is devices with interfaces that obtaininputs and translate those inputs into commands (providing feedback tothe program code). For example, a user wearing an IoT device with amotion sensor can provide inputs to the AR/VR device 640, via the IoTdevice, based on the IoT device providing motion data to the AR/VRdevice 640. As understood by one of skill in the art, the projectionitself is not necessarily an interface (although it can be if providedon a handheld in two or three dimensions), but the motions of the usercan create inputs as well as the speech of the user. Additionally, theAR/VR device itself can include monitors that interpret physicalmotions, including eye movements, as inputs. As discussed above, theseinterfaces are available as part of the AR/VR device 640. The AR/VRdevice 640 translates obtained inputs from the user and provides them tothe inventory management system 610 (603). Thus, based on these inputs(based on the projection), the program code triggers the inventorymanagement system 610 to automatically generate, in accordance with theinputs of the user, one or more of reports, related to the inventoryobject 660 and/or order for replenishment or return of the inventoryobject 660. Generating an order is one of a number of actions that theprogram code can take in embodiments of the present invention. Inembodiments of the present invention, based on the input of the user,the program code can automatically complete or initiate a variety oftasks, including but not limited to: 1) generating an order request foran item represented in the AR/VR simulation/visualization created by theprogram code executing on the device; 2) counting inventory for an itemrepresented in the AR/VR simulation/visualization created by the programcode executing on the device; 3) triggering a recall of an itemrepresented in the AR/VR simulation/visualization created by the programcode executing on the device; and/or 4) triggering a search in the oneor more designated databases for additional data to supplement thedescriptive text (generated by the program code and also displayed inthe interface) related to an item represented in the AR/VRsimulation/visualization created by the program code executing on thedevice.

In addition to the general workflow described above, FIG. 6 alsoillustrates a workflow where the object 660 is more than one item.Returning, to FIG. 6, one or more programs executing on the processor640, obtains a signal of decodable indicia from an inventory object 660(609). The one or more programs (also referred to as program code)decode the signal of decodable indicia to access decoded data (601). Thedecoded data includes information identifying the object 660. Althoughthe singular, object 660, in used, the object can include a plurality ofitems. The one or more programs then obtain, either from the memory ofthe processor 640 and/or an inventory management system 610communicatively coupled to the processor, a visual representation of aportion of the object 660 (602) (604). The one or more programs displaysthe augmented/virtual reality visual representation of the object 660(606). In some embodiments of the present invention, the program codedisplays the visual representation in three dimensions. The visualrepresentation can include a plurality of regions and each regionrepresents an item of the plurality of items (keeping in mind that theobject 660 itself is made up of more than one item, in this non-limitingexample). The visual can be either two or three dimensional. Through auser interface of the AR/VR device 630, which the one or more programsare utilizing to display the visual representation, the one or moreprograms obtain a designation of at least one of the plurality ofregions (606). The one or more programs in some embodiments of thepresent invention, based on obtaining the designation, generates anorder request for an item represented by the designated region (607). Insome embodiments of the present invention, the one or more programs makethis request through the inventory management system 610. As discussedabove, generating an order is only one example of actions that can betaken by the program code in embodiments of the present invention basedon inputs by the user.

Because embodiments of the present invention provide visual feedback tousers in real-time, in some embodiments of the present invention, basedon generating the order (or taking another action), the program codeupdates the data projection 670 to reflect the change (based on theorder or other action). Referring to FIG. 7, which will be discussed ingreater detail, as a composite image comprising the projection 780(e.g., FIG. 6, 670), includes an image 795 of the object and datadescribing the object, presented as text 790, in some embodiments of thepresent invention, based on the action, the program code can update thetext 790 to show the pending change in inventory that is a results ofthe action. For example, if the action were an order, the program codewould update the inventory. If the action were a recall request, theprogram code could update the expiration of the item.

As mentioned briefly above, FIG. 7 is an example of a composite image780 (also referred to as a data visualization and/or projection)produced by program code executing on one or more processors in someembodiments of the present invention. As illustrated in FIG. 7, theprogram code obtains various indicia (e.g., 709, 703, 702, 701, 707,which are also depicted as example with similar numbering in FIG. 5) viaa AR/VR device that includes an element capable of a projection 720 intothree dimensional space and a sensor array/processor 717, to obtain andinterpret the indicia (in order to potentially pull additional data froman inventory management system and/or other data sources with which thedevice can communicate over a communications connection). The programcode executing on the processor 717 generates the visual representation,which can include one or more of a projection 780 into three dimensionalspace and/or a representation on a handheld device 708. In someembodiments of the present invention, a user interacts with andmanipulates the projection 780 by interacting with interfaces, includingbut not limited to, a touchscreen, of the handheld device 708. Asillustrated in FIG. 7, the visual generated by the program code thatappears as a projection to the user, due to the AR/VR devicefunctionality, includes the image of the object 795 and data associatedwith the object 790. The program code obtains the data in real-time. Insome embodiments of the present invention, the image of the object 795is captured by the program code when the program code obtains theindicia, but in some embodiments of the present invention, the image isretained and obtained, based on the indicia.

Returning to FIG. 6, in some embodiments of the present invention, theprogram code obtains images from an inventory management system 610, toutilize when generating the composite image (e.g., FIG. 7, 780). In someembodiments of the present invention, either the inventory managementsystem 610 and/or the AR/VR processor 640 includes back-end data, theenables the user, through an interface, to make an image-based selectionfor inventory management (606) (603). The back-end data includes, insome embodiments of the present invention, meta-data and data, whichinclude product and scheduling data, as well as kit layer informationfor processing kits. Although this data can be stored on a server, or ina centralized database, including in the resources of the inventorymanagement system 610, the program code executing on the AR/VR device,can request a data synchronization to transfer the meta-data from thedata manager to a memory resource local to the device. The datasynchronization enables the mobile application to operate inenvironments without an easily accessible communications connection.

As discussed above, the example of replenishing items in a collection ishelpful in illustrating certain functionality of the present invention.Thus, this non-limiting example is returned to and illustrated in FIG.8. In some embodiments of the present invention, functionality relatedto the visual kit inventory described utilizes on meta-data. When usingthe visual scanning feature for kits, the meta-data supporting thisoperation includes, but is not limited to data definitions pertainingto: kit information, kit layer information, and layer products. KitInformation is data specific to a kit itself and includes, but is notlimited to, SKU numbers, kit aliases, descriptions, lot codes, and thenumber of layers present within the kit. Kit Layer Information includeskit layer meta-data which includes the high-resolution images of the kitand information designating which layer each image pertains to (e.g.,First/top, second, etc.). Layer Products meta-data provides the productID and product information for each product present on the kit layer. Aswill be understood by one of skill in the art, although the productutilized in the examples given is a medical kit commonly utilized in anOR, the techniques of the present invention can be applied to any itemsand are especially helpful when the inventory is kept in layers in agiven container, like, for example, a tool case.

FIG. 8 is an example of a kit, but displayed in two dimensions, for easeof understanding, though embodiments of the present invention cangenerate the display of FIG. 8 in three dimensions as well as in twodimensions. As seen in FIG. 8, once the kit layers, images and productshave been obtained, the program code can create the coordinatepoints/shapes for each product location on each image. Thus, returningto FIG. 7, the composite image 780 displayed by the program code caninclude the detail provided by the metadata and the image resources.

Referring to FIG. 8, in an embodiment of the present invention, whilethe program code is scanning products and/or otherwise obtaining theindicia, the data is retrieved from the database utilizing the serviceslayers. In an embodiment of the present technique, if the productscanned in a kit and the kit may be opened to audit, the system willload the associated image for the kit and layer. The user may navigatebetween layers of a kit by interacting with a virtual reproduction ofthe whole kit and/or the individual layers. As aforementioned, the threedimensional view generated by and augmented/virtual reality deviceprovides the user with a three dimensional view of the kit, but for easeof use, the device may render each layer separately and enable the userto interact with one layer at a time. When utilizing an augmentedreality device to view the kit and the layers, the user may alternativebetween layers by making pre-determined inputs into an I/O device thatis communicatively coupled to the augmented reality device. When using avirtual reality device, the user may move between various layers and/orinteract with the kit as a whole, by using movements and/or voicecommands. The device may provide haptic feedback to the user to indicatethe acceptance of the inputs.

Once a user has made a selection, an auditor can compare the picture onthe screen and select items that are not present on the kit layer. Thiswill cause the item to “Gray Out” or otherwise highlight the image. Oncethe inventory has been completed for the kit, the program codedetermines which products are present (and those that are not) andbuilds the inventory list for the kit. In some embodiments of thepresent invention, the actions of the auditor are accomplishedautomatically by the program code.

The program code produces a picture of an item upon obtaining a barcodeand/or other indicia, decoding the indicia, and determining that itrepresents a kit comprised of trays. Upon viewing a three dimensionalrepresentation of the image (generated by an augmented reality orvirtual reality device), a user can compare the rendering of a fulltray, with the actual tray s/he is examining. When the user locatesempty areas in the actual tray, the user can select the missing itemseither through a GUI and/or based on making movements and/or voicecommands. The user can select and highlight areas where items aremissing. The program code receives these selections and creates a recordof missing items for transmission to the server so that the data managercan create a replenishment order. Upon receipt of the selections, thevirtual or augmented reality device may provide the user with hapticfeedback.

As illustrated in FIG. 7, in an embodiments of the present invention,the virtual reality device or the augmented reality device utilized torender the three dimensional views 795 of the products may also displaythe product information 790 to the user. Thus, a user need not beknowledgeable about the products initially and can view this informationas s/he is making a selection. In some embodiments of the presentinvention, when a selection is made, the device may display theinformation about the product selected (as retrieved from the database)such that the user has the opportunity to review and/or confirm an itemselected. Thus, in some embodiments of the present invention, theproduct information 790 (FIG. 7) would appear as part of a compositeimage, including in the image example of FIG. 8.

For example, in embodiments of the present invention, the program code,based on obtaining the designation (e.g., of a region in FIG. 8), cangenerate an order request. However, the program code can also loginventory data for reporting and acquire data relating to theexpirations and or patient/case number for an item represented by the atleast one of the plurality of regions. The program code can store thedata acquired to memory or display the data in an AR/VR environment byinterfacing with inventory management system and/or readable indicia.The data displayed can include, but is not limited to, manufacture, itemdescription, GTIN, expiration date, date delivered, current inventorydate and count, last inventory date and count, product family, serialnumber, lot number, chain of custody, location, unique ID, recall/fieldactions, recall date, reference number, dimensional data, weight/mass,sterilization method, temperature and patient/case ID and/or otherelectronic medical record (EMR) identifier. In some embodiments of thepresent invention, as demonstrated in FIG. 7, the program code canvisually overlay the information on an image of a surgical kit, such asthat displayed in FIG. 8, as the surgical kit contains implants,surgical instruments, sterilization process department to visually alignor using other readable indicia medical supply contents with assignedstorage location compartments within kits or other storage devices.

In embodiments of the present invention, a device utilized by a user toobtain identifiable information regarding an object can utilize varioustechniques to obtain data utilized to identify the object efficiently.The device may utilize a single view of multiple objects to capturebarcodes and/or other indicia. The device may also read verbiage oncontainers, in cases where barcodes may be damaged. The device may alsocapture information in real-time. The device may be utilized torecognize an object based on its silhouette and other form factors ofits images. As discussed in FIG. 1, the device, in some embodiments ofthe present invention, reads indicia broadcasting information throughoutthe electromagnetic spectrum, including but not limited to RFID, NFC,Passive Wi-Fi, Wi-Fi, UHF, and/or VHF spectrum technologies.

In embodiments of the present invention where the augmented or virtualreality device displays one tray at a time (or in other areas,sub-images and comprise a full image of an item), upon completing ananalysis of one tray, utilizing the augmented reality or virtual realitydevice, the program code can render the next tray in the given kit. Justas with the first tray, the user can select areas to represent missingitems and thus, enable the program code that comprises the data managerto create a replenishment order for the missing items (e.g., FIG. 6,607).

Items related to the medical industry are merely offered as examples ofhow notes features of the present system work. As will be understood byone of skill in the art, the systems and methods disclosed herein areapplicable across inventory management systems in a variety ofindustries, including for example retail merchandise, automotive, foodservice, etc.

Embodiments of the present invention can be utilized to manage theinventory in an operating room (OR), for example, for inventorymanagement. Embodiments of the present invention may be utilized tocapture consumption of materials in the OR at time of completion of theOR procedure or case. A qualified user in the OR setting, including butnot limited to a Sales Rep., and/or an OR Nurse, may use thisfunctionality to capture inventory consumed during a surgical procedure.By capturing inventory use during a surgical procedure, a full-cyclecount during each service event is no longer needed to track theinventory. By utilizing this functionality, a full cycle (physical)count can occur less frequently, for example, as a monthly event or aquarterly event, with a reconciliation capability built in.

In an embodiment of the present invention, the GUI used for OR InventoryManagement includes an input (such as a button rendered on a screen), toenable a user to make an entry. Because of the time constraints of usersworking in an OR, an embodiment of the present invention utilizes theaforementioned memory resources to pre-populate data by matching on dataentries by the user. Information collected for a given surgery mayinclude, but is not limited to: Date/Time of surgery, Patient unique ID(to protect the privacy and comply with privacy regulations, embodimentsof the present invention will not track patient-specific data that wouldenable individuals using the system to identify the patients), SurgeonName, Sales Rep Name, Sales Rep ID, Manufacturer Utilized (OEM),Provider Location (may be inclusive of all locations to a particularchain). Product information tracked within an embodiment of the presentinvention may include, but is not limited to: Product ID, Product Nameand Description, and Quantity.

In an embodiment of the present invention, a user may identify whichproducts are consumed in an OR setting (or any setting) by utilizingvisuals (including augmented or virtual representations) of the kitsdisplayed by the invention. This aspect was discussed earlier, however,in an embodiment of the present invention, a user may utilize thisfeature to replenish items used in an OR setting by interacting withand/or experiencing a three dimensional rendering to select productsused from a kit (including via gestures, inputs through a GUI, and/orvoice commands). As discussed earlier, all products visually selectedmay be added to a consumption list by the present system and method.Other features of the present invention include, but are not limited to,a search feature enabling a user to search for kits (or other products)by keyword, name or product id, a select feature that enables a user toselect item on picture, wherein the software will obtain the selectionand display information about the selected product, including but notlimited to ID, Name, Description and/or Quantity.

Benefits of utilizing the described system and method for inventoryingcommercial products, including but not limited to, sterile packedhealthcare products, include: 1) increasing speed by significantlyreducing the auditor's time investment in securing the criticalinformation contained in the bar codes located on the outside of theimplant boxes; 2) increasing accuracy by reducing human error introducedwhen scanning an individual box is accidentally skipped by an auditor;and 3) increasing the ability of a user to perceive, fully, allpertinent information about a product, in a comprehensive view that doesnot interfere with other activities the user is engaged in at the time.

As discussed earlier, a benefit of embodiments of the present invention,is that it presents users with a faster and more efficient way to enterand locate information to assist the user in inventory management. Beloware some examples of areas in which embodiments of the present inventionmay be utilized.

Healthcare Areas of Use:

Total Joints

Trauma

SPD

Cranial Maxio Facial

Interventional Radiology

Cardio CRM

Electronic Medical Records—At time of input in the OR

Dental

Spine

MedSurge

Ocular

Pharmaceuticals

Hospital Supplies—Consumables

Physical Locations of Use:

Acute Care Facilities

Surgery Centers

Physician Offices

Home Healthcare

Distribution and Warehouse Centers for Healthcare products, Implants andSupplies

Medical Implant/Device Manufactures

Pharmacies

Because of the user of augmented and/or virtual reality in embodimentsof the present invention, embodiments of the present invention provide anumber of benefits to users in the area of inventory management. Forexample, various embodiments of the present invention can utilize avariety of off-the-shelf augmented reality and virtual reality devices,including many wearable devices that are hands-free when worn. Utilizinga hands-free device to view inventory-related images and/or to interactwith these images to replenish supplies, through body movements and/orvoice commands, allows described aspects of the invention to operate ina hands-free work environment. In surgical environments, such as the OR,where keeping areas sterile is of a prime concern, utilizing hands-freefunctionality assists in preserving the environment and eliminatescertain risks presented when a user must make inputs into a device withthe user's hands, as it is possible that both the device receiving theinputs and the user's hands are not sanitary. In addition to beingsanitary, a wearable device enables the mobility of the user, such thatthe user can physically navigate various areas in the environment inwhich s/he is using the device. Other advantages of utilizing a wearabledevice include, but are not limited to: comfort, durability, speed ofinput, and accuracy of input. Utilizing augmented reality and virtualreality technologies that enable ease of interaction with the software,as the multi-modal interface provides for intuitive inputs and minimizedthe requirement of product knowledge.

Embodiments of the present invention include a computer-implementedmethod, a computer program product, and a system, where program codeexecuting on one or more processors obtains a signal of decodableindicia and decodes the signal of decodable indicia to access decodeddata, where the decoded data comprises information identifying anobject, where the object comprises a plurality of items. Based on theinformation identifying the object, the program code obtain, from amemory, a visual representation of a portion of the object, where thevisual representation is divided into a plurality of regions and eachregion represents an item of the plurality of items. Based onidentifying the object, the program code obtains data comprisingdescriptive text characterizing the portion of the object, where thedescriptive text comprises quantitative inventory data related to theportion of the object. The program code displays the visualrepresentation as a three dimensional image and the descriptive text,via a device, wherein the device is selected from the group consistingof: an augmented reality device and a virtual reality device, where thethree dimensional image comprises a virtual projection in threedimensional space in a range of view of a user utilizing the device,where the device comprises a user interface. The program code obtains,via the user interface, a designation of at least one of the pluralityof regions in the visual representation. Based on obtaining thedesignation, the program code executes an action, where the actionchanges a quantitative or a qualitative element of the descriptive textfor an item represented by the at least one of the plurality of regions.The program code updates, concurrently with the executing, thedescriptive text in the visual representation to reflect the changedquantitative or qualitative element.

In some embodiments of the present invention, the action comprisesgenerating an order request for the item represented by the at least oneof the plurality of regions, and wherein the updating comprises thedescriptive text in the visual representation to reflect a change in thequantitative inventory data, based on the request for the order.

In some embodiments of the present invention, the action is selectedfrom the group consisting of: counting inventory for the itemrepresented by the at least one of the plurality of regions in one ormore designated databases communicatively coupled to the one or moreprocessors, triggering a recall of the item, triggering a search in theone or more designated databases for additional data to supplement thedescriptive text related to the item, and generating an order requestfor the item.

In some embodiments of the present invention, the action comprisescounting the inventory for the item represented by the at least one ofthe plurality of regions in the one or more designated databasescommunicatively coupled to the one or more processors, and the actionfurther comprises the program code comparing results of the counting toa number of the items in inventory at a previous time.

In some embodiments of the present invention, the interface is selectedfrom the group consisting of: a haptic feedback interface, a voicerecognition interface, an audio interface, a gesture recognitioninterface, and a biometric recognition interface.

In some embodiments of the present invention, based on obtaining thedesignation, the program code updates the displaying of the visualrepresentation to visually differentiate the at least one of theplurality of regions from other regions of the plurality of regions.

In some embodiments of the present invention, the three dimensionalimage comprises the portion of the object, as perceived by the user, inthree dimensional space.

In some embodiments of the present invention, the descriptive text isdisplayed as a virtual overlay on the object.

In some embodiments of the present invention, the program code obtains,from the memory, a visual representation of a second portion of theobject, where the visual representation of the second portion is dividedinto a plurality of regions and each region represents an item of theplurality of items, and where the second portion does not contain itemsof the plurality of items in the portion. The program code updates thethree dimensional image to include the visual representation of thesecond portion on the client. The program code obtains a seconddesignation, the second designation comprising a selection of at leastone of the plurality of regions of the visual representation of thesecond portion. Based on obtaining the second designation, the programcode updates the order request to include a request for an itemrepresented by the at least one of the plurality of regions of thevisual representation of the second portion; and the program codeupdates, concurrently with the generating, the descriptive text in thevisual representation to reflect a change in the quantitative inventorydata, based on the updated request for the order.

In some embodiments of the present invention, obtaining the signal ofdecodable indicia and the decoding the signals is executed by thedevice.

In some embodiments of the present invention, the device is furtherselected from the group consisting of a wearable device, and implanteddevice, a handheld device, and a stationary device.

In some embodiments of the present invention, the program code generatesa report based on the updated descriptive text.

In some embodiments of the present invention, the descriptive textincludes an element selected from the group consisting of: expirationdate, identifier, name, description, and quantity.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising”, when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the descriptions below, if any,are intended to include any structure, material, or act for performingthe function in combination with other elements as specifically noted.The description of the technique has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A computer-implemented method, the method comprising: obtaining, byone or more processors, a signal of decodable indicia; decoding, by theone or more processors, the signal of decodable indicia to accessdecoded data, wherein the decoded data comprises information identifyingan object, wherein the object comprises a plurality of items; based onthe information identifying the object, obtaining, by the one or moreprocessors, from a memory, a visual representation of a portion of theobject, wherein the visual representation is divided into a plurality ofregions and each region represents an item of the plurality of items;based on identifying the object, obtaining, by the one or moreprocessors, data comprising descriptive text characterizing the portionof the object, wherein the descriptive text comprises quantitativeinventory data related to the portion of the object; displaying, by theone or more processors, the visual representation as a three dimensionalimage and the descriptive text, via a device, wherein the device isselected from the group consisting of: an augmented reality device and avirtual reality device, wherein the three dimensional image comprises avirtual projection in three dimensional space in a range of view of auser utilizing the device, wherein the device comprises a userinterface; obtaining, by the one or more processors, via the userinterface, a designation of at least one of the plurality of regions inthe visual representation; based on obtaining the designation, executingan action, wherein the action changes a quantitative or a qualitativeelement of the descriptive text for an item represented by the at leastone of the plurality of regions; and updating, by the one or moreprocessors, concurrently with the executing, the descriptive text in thevisual representation to reflect the changed quantitative or qualitativeelement.
 2. The computer-implemented method of claim 1, wherein theaction comprises generating an order request for the item represented bythe at least one of the plurality of regions, and wherein the updatingcomprises the descriptive text in the visual representation to reflect achange in the quantitative inventory data, based on the request for theorder.
 3. The computer-implemented method of claim 1, wherein the actionis selected from the group consisting of: counting inventory for theitem represented by the at least one of the plurality of regions in oneor more designated databases communicatively coupled to the one or moreprocessors, triggering a recall of the item, triggering a search in theone or more designated databases for additional data to supplement thedescriptive text related to the item, and generating an order requestfor the item.
 4. The computer-implemented method of claim 3, wherein theaction comprises counting the inventory for the item represented by theat least one of the plurality of regions in the one or more designateddatabases communicatively coupled to the one or more processors, and theaction further comprises comparing, by the one or more processors,results of the counting to a number of the items in inventory at aprevious time.
 5. The computer-implemented method of claim 1, whereinthe interface is selected from the group consisting of: a hapticfeedback interface, a voice recognition interface, an audio interface, agesture recognition interface, and a biometric recognition interface. 6.The computer-implemented method of claim 1, further comprising: based onobtaining the designation, updating, by the one or more processors, thedisplaying of the visual representation to visually differentiate the atleast one of the plurality of regions from other regions of theplurality of regions.
 7. The computer-implemented method of claim 1,wherein the three dimensional image comprises the portion of the object,as perceived by the user, in three dimensional space.
 8. Thecomputer-implemented method of claim 7, wherein the descriptive text isdisplayed as a virtual overlay on the object.
 9. Thecomputer-implemented method of claim 2, further comprising: obtaining,by the one or more processors, from the memory, a visual representationof a second portion of the object, wherein the visual representation ofthe second portion is divided into a plurality of regions and eachregion represents an item of the plurality of items, and wherein thesecond portion does not contain items of the plurality of items in theportion; updating, by the one or more processors, the three dimensionalimage to include the visual representation of the second portion on theclient; obtaining, by the one or more processors, a second designation,the second designation comprising a selection of at least one of theplurality of regions of the visual representation of the second portion;based on obtaining the second designation, updating, by the one or moreprocessors, the order request to include a request for an itemrepresented by the at least one of the plurality of regions of thevisual representation of the second portion; and updating, by the one ormore processors, concurrently with the generating, the descriptive textin the visual representation to reflect a change in the quantitativeinventory data, based on the updated request for the order.
 10. Thecomputer-implemented method of claim 1, wherein the obtaining the signalof decodable indicia and the decoding the signals is executed by thedevice.
 11. The computer-implemented method of claim 1, wherein thedevice is further selected from the group consisting of a wearabledevice, and implanted device, a handheld device, and a stationarydevice.
 12. The computer-implemented method of claim 1, furthercomprising: generating, by the one or more processors, a report based onthe updated descriptive text.
 13. The computer-implemented method ofclaim 1, wherein the descriptive text includes an element selected fromthe group consisting of: expiration date, identifier, name, description,and quantity.
 14. A computer program product comprising: a computerreadable storage medium readable by one or more processors and storinginstructions for execution by the one or more processors for performinga method comprising: obtaining, by the one or more processors, a signalof decodable indicia; decoding, by the one or more processors, thesignal of decodable indicia to access decoded data, wherein the decodeddata comprises information identifying an object, wherein the objectcomprises a plurality of items; based on the information identifying theobject, obtaining, by the one or more processors, from a memory, avisual representation of a portion of the object, wherein the visualrepresentation is divided into a plurality of regions and each regionrepresents an item of the plurality of items; based on identifying theobject, obtaining, by the one or more processors, data comprisingdescriptive text characterizing the portion of the object, wherein thedescriptive text comprises quantitative inventory data related to theportion of the object; displaying, by the one or more processors, thevisual representation as a three dimensional image and the descriptivetext, via a device, wherein the device is selected from the groupconsisting of: an augmented reality device and a virtual reality device,wherein the three dimensional image comprises a virtual projection inthree dimensional space in a range of view of a user utilizing thedevice, wherein the device comprises a user interface; obtaining, by theone or more processors, via the user interface, a designation of atleast one of the plurality of regions in the visual representation;based on obtaining the designation, executing an action, wherein theaction changes a quantitative or a qualitative element of thedescriptive text for an item represented by the at least one of theplurality of regions; and updating, by the one or more processors,concurrently with the executing, the descriptive text in the visualrepresentation to reflect the changed quantitative or qualitativeelement.
 15. The computer program product of claim 14, wherein theaction comprises generating an order request for the item represented bythe at least one of the plurality of regions, and wherein the updatingcomprises the descriptive text in the visual representation to reflect achange in the quantitative inventory data, based on the request for theorder.
 16. The computer program product of claim 14, wherein the actionis selected from the group consisting of: counting inventory for theitem represented by the at least one of the plurality of regions in oneor more designated databases communicatively coupled to the one or moreprocessors, triggering a recall of the item, triggering a search in theone or more designated databases for additional data to supplement thedescriptive text related to the item, and generating an order requestfor the item.
 17. The computer program product of claim 16, wherein theaction comprises counting the inventory for the item represented by theat least one of the plurality of regions in the one or more designateddatabases communicatively coupled to the one or more processors, and theaction further comprises comparing, by the one or more processors,results of the counting to a number of the items in inventory at aprevious time.
 18. The computer program product of claim 14, wherein theinterface is selected from the group consisting of: a haptic feedbackinterface, a voice recognition interface, an audio interface, a gesturerecognition interface, and a biometric recognition interface.
 19. Thecomputer program product of claim 14, further comprising: based onobtaining the designation, updating, by the one or more processors, thedisplaying of the visual representation to visually differentiate the atleast one of the plurality of regions from other regions of theplurality of regions.
 20. A system comprising: a memory; one or moreprocessors in communication with the memory; a device selected from thegroup consisting of an augmented reality device and a virtual realitydevice, in communication with the one or more processors; programinstructions executable by the one or more processors via the memory toperform a method, the method comprising: obtaining, by the one or moreprocessors, a signal of decodable indicia; decoding, by the one or moreprocessors, the signal of decodable indicia to access decoded data,wherein the decoded data comprises information identifying an object,wherein the object comprises a plurality of items; based on theinformation identifying the object, obtaining, by the one or moreprocessors, from the memory, a visual representation of a portion of theobject, wherein the visual representation is divided into a plurality ofregions and each region represents an item of the plurality of items;based on identifying the object, obtaining, by the one or moreprocessors, data comprising descriptive text characterizing the portionof the object, wherein the descriptive text comprises quantitativeinventory data related to the portion of the object; displaying, by theone or more processors, the visual representation as a three dimensionalimage and the descriptive text, via the device, wherein the threedimensional image comprises a virtual projection in three dimensionalspace in a range of view of a user utilizing the device, wherein thedevice comprises a user interface; obtaining, by the one or moreprocessors, via the user interface, a designation of at least one of theplurality of regions in the visual representation; based on obtainingthe designation, executing an action, wherein the action changes aquantitative or a qualitative element of the descriptive text for anitem represented by the at least one of the plurality of regions; andupdating, by the one or more processors, concurrently with theexecuting, the descriptive text in the visual representation to reflectthe changed quantitative or qualitative element.